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ABSTRACT 

This paper explores the issue of the social relevance 
of instructional technology research, and examines the state of 
research in the field today. A modification of Dick and Dick's (1989) 
research article classification scheme is used to categorize the 
field's research literature. This new classification scheme 
represents an effort to distinguish between the goals of research and 
the methods of research. It is proposed that most research studies in 
instructional technology can be classified according to the six 
research goals defined in the paper: theoretical; empirical; 
interpretivist ; postmodern; developmental; and evaluation. The 
proposed methodology classification includes: quantitative; 
qualitative; critical theory; literature review; and mixed-methods. 
Matrices of research goals by research methods are provided for 104 
articles published in "Educational Technology Research and 
Development" from 1989-1994, and 129 articles published in "Journal 
of Computer-Based Instruction" from 1988-1994. The most common type 
of article in either publication is empirical in intent and 
quantitative in method. The next largest subset can be classified as 
theoretical in intent and employing literature review as the primary 
method. There is a paucity of interpretivist articles, developmental 
research studies, and a complete absence of any articles that are 
postmodern in intent or that employ critical theory as a methodology. 
The quality of contemporary research in the field is criticized, with 
many articles being characterized as pseudoscience, having the 
following characteristics: specification error; lack of linkage to 
robust theory; inadequate literature review; inadequate treatment 
implementation; measurement flaws; inconsequential outcome measures; 
inadequate sample sizes; inappropriate statistical analysis; and 
meaningless discussion of results. It is proposed that meaningful 
research can be conducted provided the sterility of existing research 
is acknowledged, and the profession builds anew from a foundation of 
sound learning theory and rededicated concern for the social impact 
of research. Data is illustrated in four figures. (Contains 54 
references.) (MAS) 
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My First Flame 

In April 1994, 1 was "flamed" on the Internet, a '90s phenomenon that has been portrayed in publications as 
diverse as The New Yorker (Seabropk, 1994) and The Chrbnicie of Higher Education (Lemisch, 1995). 
Although what exactly constitutes'a "flame'' ln''tWVap'ldty'evoTvmg '"metaverse" (Stephenson, 1993) is a 
matter of much debate, I can vividly recall the feelings of shock and anger that swept through me when I 
read the note calling me a "jerk" on a "listserv" shared by hundreds of members around the world. 

It all began last spring when I read two queries from doctoral students on the Qualitative Research for the 
Human Sciences listserv <QUALRS-L@uga.cc.uga.edu>. Both students came from large public 
institutions of higher education, one in the USA and the other in Canada. The first student wrote that she 
intended to focus her dissertation research on the quality of "discourse" that takes place in cafes and coffee 
shops located inside bookstores. She complained that she had found no "literature" on this topic and asked 
the listserv participants for some guidance. The second student announced that he was preparing a 
dissertation prospectus centered on the question of how people learned about opportunities to take SCUBA 
diving lessons and what motivated them to register for such courses. He also sought directions to relevant 
literature and advice from the listser.' membership. 

After pondering these queries, I posted a message asking whether faculty members at taxpayer-supported 
universities have a moral responsibility to guide tlieir students toward "socially responsible" research 
questions. In my posting, I suggested that in the face of problems such as adult illiteracy, attacks on public 
education, "at-risk" students, homelessness, AIDS, and the like, faculty members should attempt to inspire 
in students a dedication to research that would "make a difference." 

Soon after posting my note, the graduate student who had sought help with his SCUBA query "flamed" me 
with his "jerk" note in which he went on to criticize my "attack" on his freertom to address whatever 
research questions interested him, especially given he was a taxpayer as well. A small grass fire of flames 
then erupted as several listserv miembers castigated the student for calling me a jerk, some agreed with my 
critique, and others defended the perspective that the social relevance of doctoral dissertation research (or any 
educational research) was irrelevant. No resolution of this issue was reached on the listserv, but I was 
especially impressed by the response of an education professor from a large land grant university in the 
USA who agreed with my criticism, but went on to suggest that much of the research he has read in the 
field of instructional technology could be subjected to a similar critique. This prompted me to ponder the 
social relevancy of research in our field. 

Is Instructional Technology Research Socially Relevant? 

Social relevance is an issue that is obviously subject to much debate. One's age, race, gender, 
socioeconomic status, education, religion, political allegiance, and many other factors are likely to influence 
one's interpretation of the social relevance of any given research study. Nevertheless, for tlie sake of this 
analysis, I will attempt to define social relevance with respect to scientific inquiry. My definition is based 
upon the following principles that guide scientific research (derived from Casti, 1989): 

• Science is an ideology that consists of a cognitive structure concerning the nature of reality together with 
processes of inquiiy, verification, and peer review. 

• Views of reality differ according to one's philosophy of science, e.g., realism maintains that an objective 
reality exists, instrumentalism asserts that reality is the readings noted on measuring instruments, and 
relativism claims that reality is what the community says it is. 

• Scientific research is a social activity that has certain standards and norms, e.g., it should not 
intentionally harm humans and it must be able to be replicated by other researchers. 

• Socially responsible research in education adheres to the basic principles listed above while at the same 
time it addresses problems that detract from the quality of life for individuals and groups in society, 
especially those problems related to le.iming and human development. 

In the view of some, instructional technology research might lay claim to a blanket imprimatur with 
respect to being "socially responsible." After all, at some level, all instructional technology research can be 
said to focus on questions of how people learn and perform, especially with respect to how learning and 
performance are influenced, supported, or perhaps even caused by technology. As long as research is 
focused on learning and performance problems, and adheres to the principles listed above, it would seem to 
be socially responsible. 
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Others in the research comiTiunity argue that concern for the social responsibility of research in instructional 
technology or any other field is ludicrous. They maintain that the goal of research is knowledge in and of 
itself, and that whether research is socially responsible is a question that lies outside the bounds of science 
(cf., Carroll, 1973). In i:iy experience, researchers in the natural sciences such as biology and cheiTiistry do 
not often concern theiTiselves with the relevance question, but this is a debate that has raged for decades 
aiTiong educational researchers (see Note 1). For exaiTiple, as reported by Farley (1982), Nate Gage, a past 
president of the AiTierican Educational Research Association (AERA), has been a staunch defender of the 
notion that the goal of basic res'earch in education is simply "more valid and more positive conclusions" (p. 
12) whereas another pa.st president of AERA, Robert Ebel, proclaimed: 

....the value of basic research in education is severely limited, and here is the reason. The process of 
education is not a natural phenomenon of the kind that has sometimes rewarded scientific investigation. It 
is not one of the givens in our universe. It is man-made, designed to serve our needs. It is not governed by 
any natural laws. It is not in need of research to find out how it works. It is in need of creative invention 
to make it work better, (p. 18, Ebel's italics). 

In my opinion, Ebel's stance (with which I agree) is directly relevant to the issue of socially responsible 
research in instructional technology. There is little social releva'^ce in research studies that are largely 
focused on understanding "how" instructional technology work > without substantial concern for how this 
understanding makes educatim better. On the other hand, there is considerable social relevance in I.T. 
research studies that are largely focused on making education better (and which in the process may also help 
us understand more about how instructional technology works). 

Most of the research in instructional technology is grounded in a "realist" philosophy of science, i.e., 
conducted under the assumption that education is part of an objective reality governed by natural laws and 
therefore can be studied in a manner similar to other natural sciences such as chemistry and biology. If this 
assumption about the nature of the phenomena we study is erroneous (and I believe it is), then we 
inevitably ask the wrong questions in our research. Further, even if there arc underlying laws that influence 
learning, the complexity inherent in these laws may defy our ability to perceive, much less control, them 
(Casti, 1994). As Cronbach (1975) pointed out two decades ago, our empirical research may be doomed to 
failure because we simply cannot pile up generalizations fast enough to adapt our instructional treatments to 
the myriad of variables inherent in any given instance of instruction. 

Of course, I am not the first person to express concerns about the nature of research in instructional 
technology. I adopted the title of this paper from one published twenty-seven years ago by Keith Mielke 
(1968) titled "Questioning the Questions of ETV Research." Other critics of the questions and methods of 
re.search in instructional technology include Lumsdaine (1963), Schramm (1977), Clark (1983), and 
Salomon (1991). The debate about the nature, of reality and the conduct of research in our field continues as 
evidenced by the recent spate of articles fcKused on the question of "Does media influence learning?" (Clark, 
1994a, b; Jona.ssen, Campbell, & Davidson, 1994; Kozma, 1994a, b; Morrison, 1994; Reiser, 1994; Ross, 
1994a, b; Shrock, 1994; Tennyson, 1994; Ullmer, 1994). However, few critics have dealt directly with 
questions of whether instructional technology research is, can be, or should be socially responsible. That is 
the purpose of this paper. 

The State of In.structional Technology Research 

Before reluming to the is.sue of the social relevance of instructional technology research, it is necessary to 
examine the state of research in the field today. To accomplish this, I reviewed the contents of two of the 
primary research journals in the field, the Educational Technology Research and Development (ETR&D) 
journal and the Journal of Computer-based Instruction (JCBI) over the periods 1989-94 for ETR&D and 
1 988-93 for JCBI (see Note 2). 

For this review, I originally intended to use a research article classification scheme developed by Dick and 
Dick (1989) (see Figure 1), but my initial attempts to categorize articles using that scheme led to several 
difficulties, especially in terms of classifying studies that were primarily interpretivist ir in'^nt and 
naturalistic in method, e.g., Neuman (1991). 
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Literature review 


a summary of a body of literature, sometimes as a critique and sometimes as 
a statement of the state of the art. 


Methodological article 


a new model or procedure for carrying out a technical activity. 


Theoretical article 


one which primarily draws upon and contributes to the theoretical literature 
of the field. 


Empirical and 
experimental studies 


all studies, other than evaluations, which use data in order to draw 
conclu.sions. 


Descriptive study 


a presentation of infomiation about a particular program or event, with little 
or no use of data. 


Evaluation study 


a presentation of data and information to describe the effectiveness of a 
particular program or method, usually in an applied setting. 


Professional article 


a description of topics dealing with the profession of instructional 
technology, such as determination of competencies or descriptions of 
internship programs. 



Figure I. Research article classification scheme from Dick and Dick (1989). 

After reflection and consultations v/ith several research experts, I modified the classification scheme (see 
Note 3) This new classification scheme represents an effort to distinguish between the goals of research and 
the methods of research. First, I propose that most research studies in instructional technology can be 
classified according to the six research goals represented in Figure 2. This scheme is partially based upon 
discussions of research "paradigms" that have dominated educational research literature in recent years. For 
example, according to Soltis (1992), there are currently "three major paradigms, or three different ways of 
investigating important aspects of education" (p. 620) used in educational research: I) the positivist or 
quantitative paradigm, 2) the interpretivist or rv i.liiative paradigm, and 3) ihe critical theory or neomarxist 
paradigm. Although the "paradigm debate" literature is fascinating, I do ne t feel that the three categories 
presented by Soltis (1992) and others (e.g., Schubert & Schubert, 1990) capture the full breadth of research 
goals in the field of instructional technology. At the same time, there may be other goals of inquiry that 
are not be included in this scheme. 



Theoretical 


research focused on explaining phenomena through the logical analysis and 
synthesis of theories, principles, and the results of other forms of research 
such as empirical studies. 


Empirical 


research focused on determining how education works by testing conclusions 
related to theories of communication, learning, performance, and technology. 


Interpretivist 


research focused on portraying how education works by describing and 
interpreting phenomena related to human communication, learning, 
performance, and the use of technology. 


Postmodern 


research focused on examining the assumptions underlying applications of 
technology in human corr^iunication, learning, and performance with the 
ultimate goal of revealing hidden agendas and empowering disenfranchised 
minorities. 


Developmental 


research focused on the invention and improvement of creative approaches to 
enhancing human communication, learning, and performance through the 
use of technology and theory. 


Evaluation 


research focused on a particular program, product, or method, usually in an 
applied setting, for the purpose of describing it, improving it, or estimating 
its effectiveness and wortli. 



Figure 2. Research goal classification scheme. 
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Second, given the aforementioned desire tc separate the goals of research stu Jies from the methodologies 
employed in them, I propose the methodology classification scheme represented in Figure 3. Of course, 
there are numerous methods available to researchers in instructional technology (cf., Driscoll, 1995), but for 
the sake of simplicity, these five methodological groupings provide sufficient discrimination to allow the 
analysis represented below. 



Quantitative 


experimental, quasi-experimental, correlational, and other methods that 
primarily involve the collection of quantitative data and its analysis using 
inferential statistics. 


Qualitative 


observation, case-studies, diaries, interviews, and other methods that 
primarily involve the collection of qualitative data and its analysis using 
grounded theory and ethnographic approaches. 


Critical Theory 


deconstruction of "texts" and the technologies that deliver them through the 
search for binary oppositions, hidden agendas, and the disenfranchisement of 
minorities. 


Literature Review 


various forms of research synthesis that primarily involve the analysis and 
integration of other forms of research, e.g., frequency counts and meta- 
analyses. 


Mixed-methods 


research approaches that combine a mixture of methods, usually quantitative 
and qualitative, to triangulate findings. 



Figure 3. Research methods classification scheme. 

The combination of the goal classification and the methods classification schemes yields a matrix of 
research goals by research methods. Figure 4 presents my analysis of the research articles published in 
ETR&D (1989-1994). There were one hundred and four articles published in the research section of 
ETR&D in the six years from 1989 through 1994 (see Note 4). Not every article could be classified 
according to the classification m.atrix illustrated in Figure 4. Six "methodological articles" (presenting a 
new method or procedure for carrying out research) and three "professional articles" (analyzing the state of 
the profession of instructional technology) are not included in Figure 4. 





Quantitative 


Qualitative 
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Review 


Mixed- 
methods 


Theoretical 








31 




Empirical 


39 
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Interpretivist 
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Postmodern 












Developmental 




1 




4 


1 


Evaluation 




2 




2 


5 



Figure 4. Classification of ETR&D articles (1989-94). 

Figure 5 is an analysis of the research articles published in JCBI (1988-1993). There were one hundred and 
twenty-nine articles published in JCBI from 1988 through 1993. Five "methodological articles" and one 
"professional article" are not included in Figure 5. 
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Theoretical 








17 




Empirical 


56 
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4 


Interpretivist 
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7 


4 
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23 



Figure 5. Classification of JCBI articles (1988-93). 



O 463 

ERIC 



UJHJ!l»B!|lliJ:IIJ 



There are some obvious trends in the articles that appeared in ETR&D and JCBI during the respective 
review periods. First, the most common type of article in either publication is empirical in intent and 
quantitative in method. Thirty-nine articles (38% of the total 104) in ETR&D and fifty-six articles (43% of 
the total 129) in JCBI fall into the "empirical-quantitative" cell of the matrix. 

The next largest subset of articles in these publications can be classified as theoretical in intent and 
employing literature review as the primary method. I was liberal in my classification of articles into this 
category. For example, I assigned all of the aforementioned media debate articles into this classification 
(cf., Clark, 1994a, b; Kozma, 1994a,b). The extent to which literature review methods were actually used 
in these articles varies greatly. rr-^rs 
Another ttend that stands out is the paucity of interpretivist articles (one in ETR&D and three m JCBI) 
during this time. This seems surprising given the numerous applications of the "Constructivist- 
Hermeneutic-Interpretivist-Qualitative Paradigm" in other fields of education (cf., Eisner, 1991). Although 
Neuman (1989), Driscoll (1995), Robinson (1995) and others promote interpretivist approaches to research 
in instructional technology, interpretivist research reports rarely find their way into our publications. 

Develop ..ntal research studies are also scarce in each of these publications. With respect to ETR&D, it 
may be that most developmental research studies appear in the development section of the journal, but this 
is a hypotliesis that has not been investigated. Other possible explanations are that instructional 
technolosists rarely conduct developmental research, those that do have too little time to report it, or the 
review panels for the journals do not recognize this approach as legitimate research. 

The complete absence of any articles in these journals that are postmodern in intent or that employ critical 
theory as a methodology is disappointing, but not too surprising. First, Hlynka and Belland's (1991) 
volume on the application of postmodern criticism to instructional technology may not be widely known. 
Second, the gatekeepers of ETR&D and JCBI appear to have strong preferences for empirical research 
employing quantitative methods. They may be unwilling or unable to entertain such radical departures from 
standard research methods as have been proposed by Yeaman (1994) and other critical theorists. 

An interesting difference between the two journals is the percentage of articles that are evaluative in intent. 
Only nine (9%) of the articles in ETR&D were evaluation reports during this period whereas thirty-seven 
(29%) of the articles in JCBI were evaluations. This difference may be explained by evaluation articles in 
ETR&D being primarily relegated to the development section of the journal. As above, this hypothesis has 
not been investigated. 

The Problem of Pseudoscience 

A deeper analysis of those studies published in ETR&D and JCBI which are empirical in intent and 
quantitative in method yields a dismal picture of the quality of contemporary research in our field. In an 
earlier article published in the now defunct JCBI (Reeves, 1993), I presented an analysis of five studies 
published in refereed journals from the literature on learner control (Amone & Grabowski, 1992; Kinzie & 
Sullivan, 1989; L6pez & Harper-Meriniak, 1989; McGrath, 1992; Ross, Morrison, & O'Dell, 1989). I 
characterized the research reported in these articles as pseudoscience. Figure 6 summarizes the characteristics 
of pseudoscience in the field of instructional technology. 

Ironically, the learner control articles analyzed in Reeves (1993) are hardly the worst examples of 
pseudoscience in our field. My analysis of recent volumes of ETR&D and JCBI indicates that 
pseudoscience continues to dominate research in the field of instructional technology. A conservative 
review of the thirty -nine "empirical-quantitative" studies reported in ETR&D indicates that twenty-eight of 
them (72%) can be identified as examples of pseudoscience in that they possess two or more of the 
characteristics in Figure 6. In JCBI, thirty-four (61%) of the fifty-six "empirical-quantitative" studies 
published during this period suffer two or more signs of pseudoscience. This analysis is evidence of a 
research malaise of epidemic proportions. 
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Specification error 


Vague definitions of the primary independent variables (e.g., learner control 
versus program control). 


Lack of linkage to robust 
theory 


Little more than nominal attention to the underlying learning and instructional 
theories that are relevant to the investigation. 


Inadequate literature 
review 


Cursory literature review focused on the results of closely related studies with 
little or no consideration of alternative findings. 


Inadequate treatment 
implementation 


Infrequent (usually single) treatment implementation often averaging less than 
30 minutes. 


Measurement flaws 


Precise measurement of easy-to-measure variables (e.g., time); insufficient 
effort to establish the reliability and validity of measures of other variables. 


Inconsequential outcome 
measures 


A lack of intentionality in the learning context, usually represented by 
outcome measures that have little or no relevance for the subjects in the study. 


Inadequate sample sizes 


Small samples of convenience, e.g , the ubiquitous undergraduate teacher 
education or psychology majors. 


Inappropriate statistical 
analysis 


Use of obscure statistical procedures in an effort to tease statistically 
significant findings out of the data. 


Meaningless discussion 
of results 


Rambling, often incoherent, rationales for failing to find statistically 
significant findings. 



Figure 6. Characteristics of pseudoscience (Reeves, 1993). 



The question inevitably arises with respect to how so many pseudoscience studies get published. At least 
part of the answer rests in the incestuous nature of the relationships among the people conducting these 
studies and the people charged with peer review of these submissions. The review boards of these journals 
include many of the same people whose research studies exemplify pseudoscience. Not only does the 
insular nature of the review process assure these researchers of a venue for their pseudoscience reports, but it 
also at least partially explains the under representation of alternative approaches of inquiry. 

To understand the steady flow of pseudoscience in instructional technology, it is necessary to look at its 
source. Most of it emanates from colleges and schools of education that have graduate programs in 
instructional technology. As Kramer (1991) points out in Ed School Follies, these institutions are "intent 
on proving that education is an academic discipline with its own subject matter worthy of a place alongside 
other university schools and departments (p. 8). The faculty in these programs are subject to the same 
"publish or perish" pressure as their colleagues in arts and sciences. They quickly learn that it is the 
number of refereed publications they can amass, not the relevance or value of their research, that really 
matters when they come up for tenure and promotion. 

Needless to say, this problem is hardly limited to instructional technology programs. Colleges and schools 
of education reward pseudoscience in every discipline from eariy childhood education though vocational 
education. A new report issued by the Holmes Group called "Tomorrow's Schools of Education," calls for 
tenure and promotion guidelines to be revamped so that professors are rewarded less for research and 
publication and more for work in the public schools (Nicklin, 1995). If such a radical shift in the reward 
structure could be accomplished, I cannot believe that we would continue to conduct pseudoscience when we 
could be rewarded for making a difference in the schools where the needs are so great. 

Frankly, the likelihood of changing the reward structure within universities seems at best remote. 
However, as instructional technologists, we do not have to wait for such a change to occur. Another way 
of increasing the relevance of instructional technology would be to call a moratorium on our efforts to find 
out how instructional technology can effect learning through empirical research. Instead, we should turn 
our attention to making education work better. It seems obvious to me that we stand a better chance of 
having a positive influence on educational practice if we engage in developmental research situated in 
schools with real problems. 

Can reports of developmental research be published? Of course! After all, as noted above, the same people 
who conduct the research are the gatekeepers who determine what is accepted for publication in our most 
important journals. As academics, we are all in this together, and if we want to fundamentally change the 
nature of our "game," we can. In this manner, we can stiUmeet the frustrating, but practical, requirements 
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of the larger academic game by providing our instructional technology scholars with an outlet in refereed 
publications, albeit ones that have been radically improved in terms of goals, methods, and relevancy. 

Steps Toward Socially Responsible Research 

The relevance of pseudoscience research studies is a moot point. Even if the researchers themselves ascribe 
to the highest ideals of scientific inquiry, research so flawed has little relevance for anyone other than the 
people who conduct and publish it. But it is not enough to criticize reseai'ch in instructional technology as 
characterized by pseudoscience and social irrelevance. Alternatives to the old ways must be found. Some 
may demur, believing that instructional technologists are incapable of conducting valid, socially relevant 
research, and that they should stick to instructional design and evaluation, leaving educational research to 
cognitive psychologists or practitioners better equipped to conduct it. I disagree. I think we can and will 
conduct meaningful research provided we acknowledge the sterility of our existing research base and build 
anew from a foundation of sound learning theory and rededicated concern for the social impact of our 
research. 

What would be the nature of a new socially relevant research agenda? Two recent studies that represent a 
change in direction toward developmental research are the dissertation study conducted by Idit Harel at 
M.I.T. (1991) and the on-going research of Richard Lehrer (1993) and his associates at the University of 
Wisconsin. 

Harel's (1991) Instructional Software Design Project (ISDP) represents a unique effort to use programming 
as a cognitive tool within a software design context. Harel's ISDP combines Papert's "constructionist" 
theory (1993) with Perkins "knowledge as design" pedagogy (1986). In her dissertation research, seventeen 
fourth grade students used Logo for a semester to create software products that were intended to teach 
fractions to third grade students. Her study combined quantitative, qualitative, and comparative research 
methods to investigate the effects of this "learners as designers" approach. 

Karel reports that the fourth grade students spent an average of seventy hours working on their software 
design projects. The actual nature of the software the students designed was open, but they were two 
requirements for studei.iS in the program: 1) witing in a "Designer's Notebook" every day, and 2) attending 
periodic "Focus Sessions" about software design, Logo programming, and fractions. A teacher and the 
researcher were available at all times to help the students with their design efforts. Although each of the 
students produced a separate software product, collaboration among the students was encouraged. 

Harel compared the differences in Logo skills and fractions knowledge between the seventeen students in the 
ISDP and thirty-four other students in two classes who were studying Logo and fractions via "a traditional 
teaching method" (p. 263). No significant differences were found in pretests among tlie three classes. Harel 
reports diat "In general, the 17 children of the experimental class did better than die other 34 children on all 
posttests (Fractions and Logo)" (p. 272). Although not all differences were statistically significant, the 
general trend was quite positive in terms of specific learning outcomes as measured by multiple measures 
including paper-and-pencil tests, computer exercises, video-taped observations, and interviews. 

The major part of Harel's (1991) study is a detailed description of the activities and metacognition of one 
student, "Debbie," over the four mondi period of the project. Harel wrote that her detailed analysis of 
Debbie's work as well as her observations of other students indicated that "Throughout ISDP, the students 
were constantly involved in metacognitive acts: learning by explaining, creating, and discussing knowledge 
representations, finding design strategies, and reflecting on all of die above" (p. 359). In addition to 
positive cognitive effects in terms of metacognition, Harel concluded that the ISDP students acquired 
enhanced cognitive flexibility, better control over their problem-solving, and greater confidence in their 
thinking abilities. She notes however that the study did not include any direct measures of thinking skills, 
but is grounded in her own interpretations of the students' metacognition and problem-solving processes 
based upon observations and analysis of documentadon such as their Designer's Notebooks. 

Lehrer (1993) describes the development, use, and results of a hypermedia construction tool called 
HyperAuthor that eighth graders used to design their own lessons about the American Civil War. This 
approach is based upon the cognitive learning theory that knowledge is a process of design and not 
something to be transmitted from teacher to student (Perkins, 1986). Lehrer's students were engaged in 
"hyper-composidon" by designing their own hypermedia. In this mode, learners transform information into 
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dimensional representations, determine what is important and what is not, segment information into nodes, 
link the information segments by semantic relationships, and decide how to represent ideas. This is a 
highly motivating process because authorship results in ownership of the ideas in the hypermedia 
(Jonassen, in press). 

Lehrer's subjects were high and low ability eighth graders who worked at the hypermedia construction tasks 
for one class pc riod of 45 minutes each day over a period of several months. The students worked in the 
school's media center where they had access to a color Macintosh computer, scanner, sound digitizer, 
HyperAuthor software, and print and non-print resources about the Civil War. An instructor was available 
to coach students in the conceptualization, design, and production of hypermedia. Students created programs 
reflecting their unique interests and individual differences. For example, they created hypermedia about the 
role of women in the Civil War, the perspectives of slaves toward the war, and "not-so-famous people" 
from that period. 

According to Lehrer, "The most striking finding was the degree of student involvement and engagement" (p. 
209). Both high and low ability students became very task-oriented, increasingly so as they gained more 
autonomy and confidence with the cognitive tools. At the end of the study, students in the hypermedia 
group and a control group of students who had studied the Civil War via traditional classroom methods 
during the same period of time were given an identical teacher-constructed test of knowledge. No significa^it 
test differences were found. Lehrer conjectured that "these measures were not valid indicators of the extent 
of learning in the hypermedia design groups, perhaps because much of what students developed in the design 
context was not anticipated by the classroom teacher" (p. 218). 

However, a year later, when students in the design and control groups were interviewed by an independent 
interviewer unconnected with the previous year's work, important differences were found. Students in the 
control group could recall almost nothing about the historical content, whereas students in the design group 
displayed elaborate concepts and ideas that they had extended to other areas of history, Most importantly, 
although students in the control group defined history as the record of the facts of the past, students in the 
design class defined history as a process of interpreting the pa.st from different perspectives. In short, the 
hypermedia "design approach lead to knowledge that was richer, better connected, and more applicable to 
subsequent learning and events" (p. 221). 

A New Beginning 

What a contrast exists between the Harel (1991) and Lehrer (1993) studies and the inorass of pseudoscience 
endemic in our field! In the first instances, pedagogical models grounded in robust learning theories have 
been identified, and subsequently, powerful technologies have been used to implement these models. In the 
latter, the power of various forms of technology to instruct is assumed, and reductionist experiments are 
conducted to detect its effects. In Harel's study, children with authentic needs experienced a powerful 
learning opportunity over a period of months. In most pseudoscience studies, undergraduates earn "extra 
credit" for less than an hour of their time spent using some form of mediated "treatment" that has little or 
no relevance for them. 

In a landmark paper about educational research, Salomon (1991) de.scribes the contrast between analytic and 
systemic approaches to research. Salomon claims that this contrast transcends the "basic versus applied" or 
"quantitative versus qualitative" arguments that so often dominate debates about the relevancy of educational 
research. Salomon concludes that the analytic and systemic approaches are complementaiy, arguing that 
"the analytic approach capitalizes on precision while the systemic approach capitalizes on authenticity" (p. 



While I agree with Salomon in theory, the dominance of pseudoscience in I.T. invalidates this 
complementarity in practice. The ugly truth is that many of us who engage in analytic research approaches 
consistently violate many of the basic premises of this paradigm, especially with respect to the testing of 
meaningful hypotheses derived from strong theory (Reeves, 1993). Although we may eventually be able to 
conduct valid, socially responsible analytic studies in this field, that time has not yet arrived. We need a 
valid body of systemic (interpretivist, postmodern, and developmental) research before we begin to have 
theoretical foundations strong enough to pursue an analytical agenda. 
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Is instructional technology research socially responsible? At the present time, it is not. Are we asking the 
wrong questions? For the most part, yes. Can we change this sad state of affairs? Of course, if we have 
the will! Salomon (1991) points the way. A major benefit of systemic research in education is that it 
yields new questions and nurtures the development of new theory. The aforementioned moratorium on 
analytic studies in our field could give us the theoretical foundations for a socially relevant analytic research 
agenda early in the 21st Century. There are hopeful signs as indicated by the studies of Harel (1991) and 
Lehrer (1993) and the methodological prescriptions of Neuman (1989), Newman (1990), and Salomon 



I believe that the research malaise in I.T. stems in part from the "mindlessness" that is endemic in so much 
of our professional and personal lives as we near the 21st Century. Although some would attempt to 
redirect or revive the pseudoscience approach to I.T. research (cf., Ross & Morrison, 1989), it is clear that 
much of our research continues to sufter from the same mindless misconceptions and irrelevance identified 
by previous critics (Lumsdaine, 1963; Mielke, 1968; Schramm, 1977; Clark, 1983; Salomon, I99I). The 
social psychologist, Ellen Langer, documents the teirible costs of mindless behavior in education, health 
care, and business in her book 1989 book. Mindfulness. She writes: 

When we are behaving mindlessly, that is to say, relying on categories drawn in the past, endpoints to 
development seem fixed. We are then like projectiles moving along a predetermined course. When we are 
mindful, we see all sorts of choices and generate new endpoints. Mindful involvement in each episode of 
development makes us freer to map our own course, (pp. 96-97) 

Tne demise of JCBI, the recurring "influence of media" debate, and the prevalence of pseudoscience in our 
field are all signals that we need to become more mindful about our research. If we continue as before, 
mindlessly conducting pseudoscience, the obsolescence of our field per se is a likely outcome. Already, the 
most exciting learning and performance environments are not coming out of Departments of Instructional 
Technology (cf., Cognition and Technology Group at Vanderbilt, 1992; Gery, 1995). On the other hand, as 
Langer emphasizes, mindfulness opens up all kinds of possibilities. Let us seize this opportunity to stop 
being pawns in "someone else's costly construction of reality" (p. 28) and realize that we, and we alone, can 
assure the validity and social relevance of research in instructional technology. 
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THOMAS C. REEVES, Ph.D., is a professor of instructional technology at The University of Georgia where 
he teaches evaluation and multimedia design. Since receiving his Ph.D. at Syracuse University in 1979, he 
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Notes 

1 . The belief that biologists and other natural scientists don't have to be as concerned about the social 
relevance of their research as social scientists is being tested in the courts (Leatherman, 1995). A female 
biology professor denied tenure at Vassar College sued on the grounds that the research of the male 
professors who voted on her tenure decision was less important than hers. The judge agreed fmding that her 
research on skin differentiation might have "important implications" for cancer research whereas the research 
of the one of her male colleagues on spider behavior was "nanow" and subject to ridicule (p. A 14). 

2. Although I would have preferred to examine research publications in both journals during an identical 
six year period, this was not possible. ETR&D began its new format in 1989, and JCBI ceased publication 
at the end of 1993. 

3. I am grateful to Marcy DriscoU, Don Ely, Kent Gustafson, Mike Hannatln, John Hedberg, and Walter 
Dick for their generous guidance in the development of this revised classification scheme. Of course, I take 
full responsibility for the flaws that will no doubt be revealed in its organization. 

4. ETR&D is a product of the integration of two journals previously published by the Association for 
Educational Communications and Technology {Educational Communications and Technology Journal and 



(1991). 
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Journal of Instructional Development). ETR&D is divided into two sections, a research section and a 
development section. This analysis only considered the research section of ETR&D. 
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